Using prediction models for scene difficulty in vehicle routing

ABSTRACT

A route is selected for travel by an autonomous vehicle based on at least a level of difficulty of traversing the driving environment along that route. Vehicle signals, provided by one or more autonomous vehicles, indicating a difficulty associated with traveling a portion of a route are collected and used to predict a most favorable driving route for a given time. The signals may indicate a probability of disengaging from autonomous driving mode, a probability of being stuck for an unduly long time, traffic density, etc. A difficulty score may be computed for each road segment of a route, and then the scores of all of the road segments of the route are added together. The scores are based on number of previous disengagements, previous requests for remote assistance, unprotected left or right turns, whether parts of the driving area are occluded, etc. The difficulty score is used to compute a cost for a particular route, which may be compared to costs computed for other possible routes. Based on such information, a route may be selected.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/843,223, filed Dec. 15, 2017, the entire disclosure of which isincorporated by reference herein.

BACKGROUND

Current navigation systems consider factors such as distance and drivingtime. However, such systems provide only limited options to drivers andpassengers of vehicles, and particularly autonomous vehicles. Suchautonomous vehicles may operate in a fully autonomous mode wherepassengers may provide some initial input, such as a pickup ordestination location, and the vehicle maneuvers itself to that location.In other instances the vehicles may operate in a semi-autonomous ormanual mode, where a human driver controls some or all aspects of thevehicle maneuvering.

BRIEF SUMMARY

One aspect of the disclosure provides a method of determining a route oftravel for an autonomous vehicle. This method includes receiving, at oneor more computing devices, data from one or more vehicles, the dataproviding information regarding a road segment traversed by the one ormore vehicles, and the data including one or more difficulty signalsfrom the one or more vehicles. The data is used to determine adifficulty value for the segment, the difficulty value based ondifficulty signals from the one or more vehicles indicating disengagingfrom autonomous mode to a manual mode. In other examples, the difficultysignals may indicate requesting assistance from a remote operator, anumber of unprotected turns along the road segment, a portion of adriving area where a field of view of the autonomous vehicle isoccluded, a threshold or abnormal number of obstacles, or the like. Aroute cost is computed for each of one or more possible routes betweenan origin and a destination, wherein the route cost is based on at leastthe difficulty values for the road segments along that route, and it isdetermined which of the one or more possible routes to travel based onthe route cost for each of the one or more possible routes. In someexamples, the method further includes causing the autonomous vehicle todrive the determined route. The difficulty signals may be correlatedwith temporal features. For example, computing the difficulty value fora particular segment may include comparing a day or time of expectedtravel along the particular segment with a day or time correlated withthe difficulty signals for the particular segment. Determining thedifficulty value may include determining a probability of encountering adifficulty corresponding to at least one of the difficulty signals.

In some examples, the method may further include computing a segmentcost for the road segment, wherein the segment cost is computed based onthe difficulty value, distance, and time required to traverse the roadsegment, wherein the route cost is computed based on the segment costfor the road segments along the route. In such examples, computing thesegment cost may include computing a difficulty score based on thedifficulty value for the segment, computing a distance score based onthe distance of the segment, computing a time score based on the timerequired to traverse the segment, weighting the difficulty score, thedistance score, and the time score, and computing the segment cost as afunction of the weighted scores. Weighting the difficulty score, thedistance score, and the time score may include setting coefficients forthe difficulty score, the distance score, and the time score, whereinthe coefficients are modifiable to adjust relative weights. Input may bereceived from a passenger of the autonomous vehicle, and thecoefficients may be adjusted based on the received input from thepassenger.

Another aspect of the disclosure provides a system for determining aroute of travel for an autonomous vehicle. The system includes one ormore memories, and one or more processors in communication with the oneor more memories. The one or more processors may be configured toreceive data from one or more vehicles, the data providing informationregarding a road segment traversed by the one or more vehicles andincluding difficulty signals from the one or more vehicles, anddetermine, based on the data, a difficulty value for the segment, thedifficulty value based on difficulty signals from the one or morevehicles indicating disengaging from autonomous mode to a manual mode.In some examples, the difficulty signal may alternatively oradditionally indicate requesting assistance from a remote operator, anumber of unprotected turns along the road segment, a portion of adriving area where a field of view of the autonomous vehicle isoccluded, a number of obstacles on the road segment, etc. The one ormore processors may further compute a route cost for each of one or morepossible routes between an origin and a destination, wherein the routecost is based on at least the difficulty values for the road segmentsalong that route, and determine, based on the route cost for each of theone or more possible routes, which of the one or more possible routes totravel.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional diagram of an example vehicle in accordance withan exemplary embodiment.

FIG. 2 is an example of map information in accordance with aspects ofthe disclosure.

FIG. 3 is an example external view of a vehicle in accordance withaspects of the disclosure.

FIG. 4 is a pictorial diagram of an example system in accordance with anexemplary embodiment.

FIG. 5 is a functional diagram of the system of FIG. 4 in accordancewith aspects of the disclosure.

FIG. 6 is an example data structure in accordance with aspects of thedisclosure.

FIG. 7 is an example flow diagram in accordance with aspects of thedisclosure.

FIG. 8 is an example of possible route options associated with the mapinformation of FIG. 2.

FIG. 9 is an example of road segments for the possible route options ofFIG. 8.

FIG. 10 is an example representation of difficulty levels associatedwith the road segments of FIG. 9.

FIG. 11 is an example computation of route costs for the possible routeoptions of FIG. 8.

DETAILED DESCRIPTION Overview

The technology relates generally to using collected data to improvesystem behavior of an autonomous vehicle. In particular, such collecteddata can be included in a prediction model and used to determine a routeto take based on time, distance, and difficulty of traveling the route.

Metrics are learned from a number of vehicles, which may includeautonomous, semi-autonomous, and manually operated vehicles. Thesemetrics are collected and considered in combination. Such metrics mayinclude, for example, distance, time, and difficulty. The difficultymetric considers factors such as how difficult it was to make aparticular turn, whether there are any obstacles such as construction, anumber of pedestrian crossings, whether the roadway is rough, narrow,etc. The metrics are learned automatically from logs of previous trips,thus allowing the system to improve itself over time.

Such metrics may be associated with particular features, such astemporal features (e.g., hour of day, day of week), geographic features(e.g., precise location), and generalizing features (e.g., speed limit,nearby features such as crosswalks, intersections, schools). Forexample, while a particular turn may be difficult during a timecorresponding to morning rush hour, the same turn may be less difficultin early afternoon and far less difficult late at night. By associatingsuch features with the metrics, such features may be considered whendetermining an optimal route.

The difficulty metric may be determined based on signals indicating howdifficult it is for a vehicle to traverse a particular road segment.Such signals may indicate a probability of disengaging from autonomousdriving mode, a probability of being stuck for an unduly long time,traffic density, and the like. According to some examples, a difficultyscore is computed for each road segment, and then the scores of all ofthe road segments of a route are added together. The scores may be basedon number of previous disengagements, previous requests for assistancefrom a remote operator, unprotected left or right turns, whether partsof the driving area are occluded, user complaints, measured discomfortof passengers, fault responses in which the vehicle can no longeroperate normally and need to pull over and stop, etc.

Scores may also be computed for each segment of a route based on anexpected amount of time to complete travel along the segment, and adistance of the segment. These scores may be integrated with thedifficulty score. For example, difficulty, time, and distance scores mayall be used to compute a cost for the road segment. Such cost may beentered into a prediction model and used to determine the best route.

When determining the best route, dynamic weights can be used to adjustvarious trade-offs in routing, such as trade-offs between additionaldriving distance/time and easier driving conditions. Just as one exampleof a trade-off, a driver may be willing to drive some distance fartherto avoid making a difficult left turn. The dynamic weights may be, forexample, coefficients, which can be adjusted depending on the scenario.For example, in a first scenario where a customer is in the vehicle, thecoefficients may be adjusted to be more conservative, such as providingfor an easier though possible longer route. On the other hand, in asecond scenario where a test driver is in the vehicle, the coefficientsmay be adjusted to be more adventurous, such as providing for a quickerand more difficult route.

Using the above-described model, travel routes may be selected based onnot only time and distance, but also difficulty. Because this difficultyinformation is collected from a number of vehicle logs, it is accurateand also does not place a burden on a user for input.

Example Systems

As shown in FIG. 1, a vehicle 100 in accordance with one aspect of thedisclosure includes various components. While certain aspects of thedisclosure are particularly useful in connection with specific types ofvehicles, the vehicle may be any type of vehicle including, but notlimited to, cars, trucks, motorcycles, buses, recreational vehicles,etc. The vehicle may have one or more computing devices, such ascomputing devices 110 containing one or more processors 120, memory 130and other components typically present in general purpose computingdevices.

The memory 130 stores information accessible by the one or moreprocessors 120, including instructions 134 and data 132 that may beexecuted or otherwise used by the processor 120. The memory 130 may beof any type capable of storing information accessible by the processor,including a computing device-readable medium, or other medium thatstores data that may be read with the aid of an electronic device, suchas a hard-drive, memory card, ROM, RAM, DVD or other optical disks, aswell as other write-capable and read-only memories. Systems and methodsmay include different combinations of the foregoing, whereby differentportions of the instructions and data are stored on different types ofmedia.

The instructions 134 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor. For example, the instructions may be stored as computingdevice code on the computing device-readable medium. In that regard, theterms “instructions” and “programs” may be used interchangeably herein.The instructions may be stored in object code format for directprocessing by the processor, or in any other computing device languageincluding scripts or collections of independent source code modules thatare interpreted on demand or compiled in advance. Functions, methods androutines of the instructions are explained in more detail below.

The data 132 may be retrieved, stored or modified by processor 120 inaccordance with the instructions 134. For instance, although the claimedsubject matter is not limited by any particular data structure, the datamay be stored in computing device registers, in a relational database asa table having a plurality of different fields and records, XMLdocuments or flat files. The data may also be formatted in any computingdevice-readable format.

The one or more processor 120 may be any conventional processors, suchas commercially available CPUs. Alternatively, the one or moreprocessors may be a dedicated device such as an ASIC or otherhardware-based processor. Although FIG. 1 functionally illustrates theprocessor, memory, and other elements of computing devices 110 as beingwithin the same block, it will be understood by those of ordinary skillin the art that the processor, computing device, or memory may actuallyinclude multiple processors, computing devices, or memories that may ormay not be stored within the same physical housing. For example, memorymay be a hard drive or other storage media located in a housingdifferent from that of computing devices 110. Accordingly, references toa processor or computing device will be understood to include referencesto a collection of processors or computing devices or memories that mayor may not operate in parallel.

Computing devices 110 may include all of the components normally used inconnection with a computing device such as the processor and memorydescribed above as well as a user input 150 (e.g., a mouse, keyboard,touch screen and/or microphone) and various electronic displays (e.g., amonitor having a screen or any other electrical device that is operableto display information). In this example, the vehicle includes aninternal electronic display 152 as well as one or more speakers 154 toprovide information or audio visual experiences. In this regard,internal electronic display 152 may be located within a cabin of vehicle100 and may be used by computing devices 110 to provide information topassengers within the vehicle 100.

Computing devices 110 may also include one or more wireless networkconnections 156 to facilitate communication with other computingdevices, such as the client computing devices and server computingdevices described in detail below. The wireless network connections mayinclude short range communication protocols such as Bluetooth, Bluetoothlow energy (LE), cellular connections, as well as various configurationsand protocols including the Internet, World Wide Web, intranets, virtualprivate networks, wide area networks, local networks, private networksusing communication protocols proprietary to one or more companies,Ethernet, WiFi and HTTP, and various combinations of the foregoing.

In one example, computing devices 110 may be control computing devicesof an autonomous driving computing system or incorporated into vehicle100. The autonomous driving computing system may capable ofcommunicating with various components of the vehicle in order to controlthe movement of vehicle 100 according to primary vehicle control code ofmemory 130. For example, returning to FIG. 1, computing devices 110 maybe in communication with various systems of vehicle 100, such asdeceleration system 160, acceleration system 162, steering system 164,signaling system 166, navigation system 168, positioning system 170,perception system 172, and power system 174 (i.e. the vehicle's engineor motor) in order to control the movement, speed, etc. of vehicle 100in accordance with the instructions 134 of memory 130. Again, althoughthese systems are shown as external to computing devices 110, inactuality, these systems may also be incorporated into computing devices110, again as an autonomous driving computing system for controllingvehicle 100.

As an example, computing devices 110 may interact with one or moreactuators of the deceleration system 160 and/or acceleration system 162,such as brakes, accelerator pedal, and/or the engine or motor of thevehicle, in order to control the speed of the vehicle. Similarly, one ormore actuators of the steering system 164, such as a steering wheel,steering shaft, and/or pinion and rack in a rack and pinion system, maybe used by computing devices 110 in order to control the direction ofvehicle 100. For example, if vehicle 100 is configured for use on aroad, such as a car or truck, the steering system may include one ormore actuators to control the angle of wheels to turn the vehicle.Signaling system 166 may be used by computing devices 110 in order tosignal the vehicle's intent to other drivers or vehicles, for example,by lighting turn signals or brake lights when needed.

Navigation system 168 may be used by computing devices 110 in order todetermine and follow a route to a location. In this regard, thenavigation system 168 and/or data 132 may store detailed mapinformation, e.g., highly detailed maps identifying the shape andelevation of roadways, lane lines, intersections, crosswalks, speedlimits, traffic signals, buildings, signs, real time trafficinformation, vegetation, or other such objects and information.

FIG. 2 is an example of map information 200 for a geographic area, whichincludes a plurality of roadways. The map information 200 may includeinformation identifying the shape, location, and other characteristicsof features such as lane lines 210, 212, 214, traffic signal lights 220,222, stop signs 250, 251, 252, and other features not shown, such ascrosswalks, sidewalks, and yield signs 260. The map information 200 mayfurther include distance information for each of the plurality ofroadways, and regulations for the various roadways, such as speedlimits, prohibition of maneuvers, or the like. Just as one example of aprohibited maneuver, vehicles may be prevented from turning right on ared light between 8 am-4 pm weekdays. According to some examples, themap information may also include traffic information, such as historicaltraffic information. Such traffic information may be stored inassociation with temporal limitations, for example, indicating times ofdays/weeks when traffic is heavier.

Although the map information is depicted herein as an image-based map,the map information need not be entirely image based (for example,raster). For example, the map information may include one or moreroadgraphs or graph networks of information such as roads, lanes,intersections, and the connections between these features. Each featuremay be stored as graph data and may be associated with information suchas a geographic location and whether or not it is linked to otherrelated features, for example, a stop sign may be linked to a road andan intersection, etc. In some examples, the associated data may includegrid-based indices of a roadgraph to allow for efficient lookup ofcertain roadgraph features.

Positioning system 170 may be used by computing devices 110 in order todetermine the vehicle's relative or absolute position on a map or on theearth. For example, the position system 170 may include a GPS receiverto determine the device's latitude, longitude and/or altitude position.Other location systems such as laser-based localization systems,inertial-aided GPS, or camera-based localization may also be used toidentify the location of the vehicle. The location of the vehicle mayinclude an absolute geographical location, such as latitude, longitude,and altitude as well as relative location information, such as locationrelative to other cars immediately around it which can often bedetermined with less noise that absolute geographical location.

The positioning system 170 may also include other devices incommunication with computing devices 110, such as an accelerometer,gyroscope or another direction/speed detection device to determine thedirection and speed of the vehicle or changes thereto. By way of exampleonly, an acceleration device may determine its pitch, yaw or roll (orchanges thereto) relative to the direction of gravity or a planeperpendicular thereto. The device may also track increases or decreasesin speed and the direction of such changes. The device's provision oflocation and orientation data as set forth herein may be providedautomatically to the computing devices 110, other computing devices andcombinations of the foregoing.

The perception system 172 also includes one or more components fordetecting objects external to the vehicle such as other vehicles,obstacles in the roadway, traffic signals, signs, trees, etc. Forexample, the perception system 172 may include lasers, sonar, radar,cameras and/or any other detection devices that record data which may beprocessed by computing device 110. In the case where the vehicle is apassenger vehicle such as a minivan, the minivan may include a laser orother sensors mounted on the roof or other convenient location. Forinstance, FIG. 3 is an example external view of vehicle 100. In thisexample, roof-top housing 310 and dome housing 312 may include a lidarsensor as well as various cameras and radar units. In addition, housing320 located at the front end of vehicle 100 and housings 330, 332 on thedriver's and passenger's sides of the vehicle may each store a lidarsensor. For example, housing 330 is located in front of driver door 360.Vehicle 100 also includes housings 340, 342 for radar units and/orcameras also located on the roof of vehicle 100. Additional radar unitsand cameras (not shown) may be located at the front and rear ends ofvehicle 100 and/or on other positions along the roof or roof-top housing310.

The computing devices 110 may control the direction and speed of thevehicle by controlling various components. By way of example, computingdevices 110 may navigate the vehicle to a destination locationcompletely autonomously using data from the detailed map information andnavigation system 168. Computing devices 110 may use the positioningsystem 170 to determine the vehicle's location and perception system 172to detect and respond to objects when needed to reach the locationsafely. In order to do so, computing devices 110 may cause the vehicleto accelerate (e.g., by increasing fuel or other energy provided to theengine by acceleration system 162), decelerate (e.g., by decreasing thefuel supplied to the engine, changing gears, and/or by applying brakesby deceleration system 160), change direction (e.g., by turning thefront or rear wheels of vehicle 100 by steering system 164), and signalsuch changes (e.g., by lighting turn signals of signaling system 166).Thus, the acceleration system 162 and deceleration system 160 may be apart of a drivetrain that includes various components between an engineof the vehicle and the wheels of the vehicle. Again, by controllingthese systems, computing devices 110 may also control the drivetrain ofthe vehicle in order to maneuver the vehicle autonomously.

Computing device 110 of vehicle 100 may also receive or transferinformation to and from other computing devices, such as to othervehicles, control centers, databases, user devices, etc. FIGS. 4 and 5are pictorial and functional diagrams, respectively, of an examplesystem 400 that includes a plurality of computing devices 410, 420, 430,440 and a storage system 450 connected via a network 460. System 400also includes vehicle 100, and vehicles 100A, 100B which may beconfigured the same as or similarly to vehicle 100. Although only a fewvehicles and computing devices are depicted for simplicity, a typicalsystem may include significantly more.

As shown in FIG. 4, each of computing devices 410, 420, 430, 440 mayinclude one or more processors, memory, data and instructions. Suchprocessors, memories, data and instructions may be configured similarlyto one or more processors 120, memory 130, data 132, and instructions134 of computing device 110.

The network 460, and intervening nodes, may include variousconfigurations and protocols including short range communicationprotocols such as Bluetooth, Bluetooth LE, the Internet, World Wide Web,intranets, virtual private networks, wide area networks, local networks,private networks using communication protocols proprietary to one ormore companies, Ethernet, WiFi and HTTP, and various combinations of theforegoing. Such communication may be facilitated by any device capableof transmitting data to and from other computing devices, such as modemsand wireless interfaces.

In one example, one or more computing devices 110 may include one ormore server computing devices having a plurality of computing devices,e.g., a load balanced server farm, that exchange information withdifferent nodes of a network for the purpose of receiving, processingand transmitting the data to and from other computing devices. Forinstance, one or more computing devices 410 may include one or moreserver computing devices that are capable of communicating withcomputing device 110 of vehicle 100 or a similar computing device ofvehicle 100A, 100B as well as computing devices 420, 430, 440 via thenetwork 460. For example, vehicles 100, 100A, 100B may be a part of afleet of vehicles that can be dispatched by server computing devices tovarious locations. In this regard, the server computing devices 410 mayfunction as a dispatching system. In addition, the vehicles of the fleetmay periodically send the server computing devices location informationprovided by the vehicle's respective positioning systems as well asother information relating to the status of the vehicles discussedfurther below, and the one or more server computing devices may trackthe locations and status of each of the vehicles of the fleet.

In addition, server computing devices 410 may use network 460 totransmit and present information to a user, such as user 422, 432, 442on a display, such as displays 424, 434, 444 of computing devices 420,430, 440. In this regard, computing devices 420, 430, 440 may beconsidered client computing devices.

As shown in FIG. 4, each client computing device 420, 430, 440 may be apersonal computing device intended for use by a user 422, 432, 442, andhave all of the components normally used in connection with a personalcomputing device including a one or more processors (e.g., a centralprocessing unit (CPU)), memory (e.g., RAM and internal hard drives)storing data and instructions, a display such as displays 424, 434, 444(e.g., a monitor having a screen, a touch-screen, a projector, atelevision, or other device that is operable to display information),and user input devices 426, 436, 446 (e.g., a mouse, keyboard,touchscreen or microphone). The client computing devices may alsoinclude a camera for recording video streams, speakers, a networkinterface device, and all of the components used for connecting theseelements to one another.

Although the client computing devices 420, 430, and 440 may eachcomprise a full-sized personal computing device, they may alternativelycomprise mobile computing devices capable of wirelessly exchanging datawith a server over a network such as the Internet. By way of exampleonly, client computing device 420 may be a mobile phone or a device suchas a wireless-enabled PDA, a tablet PC, a wearable computing device orsystem, or a netbook that is capable of obtaining information via theInternet or other networks. In another example, client computing device430 may be a wearable computing system, shown as a wristwatch as shownin FIG. 4. As an example the user may input information using a smallkeyboard, a keypad, microphone, using visual signals with a camera, or atouch screen.

As with memory 130, storage system 450 can be of any type ofcomputerized storage capable of storing information accessible by theserver computing devices 410, such as a hard-drive, memory card, ROM,RAM, DVD, CD-ROM, write-capable, and read-only memories. In addition,storage system 450 may include a distributed storage system where datais stored on a plurality of different storage devices which may bephysically located at the same or different geographic locations.Storage system 450 may be connected to the computing devices via thenetwork 460 as shown in FIGS. 4 and 5, and/or may be directly connectedto or incorporated into any of the computing devices 110, 410, 420, 430,etc.

Storage system 450 may store various types of information as describedin more detail below. This information may be retrieved or otherwiseaccessed by a server computing device, such as one or more servercomputing devices 410, in order to perform some or all of the featuresdescribed herein. As an example, the information of storage system 450may include reports from other vehicles regarding difficultiesencountered during previous drives. Such difficulties may include, forexample, obstructed sensors views, obstructed roadways, unprotectedturns, rough terrain, a need for a human driver to take over control ofthe vehicle, requests for assistance, or any other difficulties. Thereports of such difficulties may be stored in association with metadatawhich identifies what the particular difficulty was and a time thedifficulty occurred. In some examples, the reports may further be storedin association with a value indicating a level of severity of thedifficulty.

FIG. 6 provides just one example of the type and arrangement ofinformation that may be stored in the storage system 450. As shown,chart 600 includes a number of columns listing vehicle signalidentifiers 610, signal type 620, timestamps 630, and location 640. Itshould be understood that additional information may also be listed, andthat some of the columns may be omitted. Just as one example, the chart600 may also store an associated difficulty value, for each signal,indicating a level of difficulty imposed by the driving environment. Itshould further be understood that the columns and rows are the chart 600may be sorted in any order, or that a different organizational structuremay be used altogether.

In the example chart 600, the vehicle signal identifier 610 may identifya specific signal sent by a vehicle. Each unique signal may have aunique identifier. This identifier may be used to distinguish particularsignals and to facilitate identification of particular information. Thesignal identifier may be included with the signal sent by the vehicle,assigned by the server computing device 410, or composed from somecombination of information from the vehicle, the serer 410, or othersources. The signals in the chart 600 may be from one vehicle or from aplurality of different vehicles.

The signal type 620 indicates a type of difficulty encountered by thevehicle when sending the signal. A few example types of difficulties arelisted in the chart 600, including a need for a human driver to takeovercontrol of the vehicle manually, a request for assistance from a remoteoperator, an extended wait (e.g., to make a turn or other maneuver), anunprotected turn, an occluded sensor view, a rapid brake application, alimited sensor view, and an unprotected lane crossing. Further examplesinclude user complaints, measured discomfort of passengers, faultresponses, etc. Other types of signals indicating a driving difficultyresulting from a driving environment may also be stored, and multiplesignals of the same type may be stored.

The timestamps 630 may indicate a time the signal was sent by thevehicle. This information may be particularly useful for difficultiesthat are time dependent. For example, it may be determined from thesignals that extended wait times to make a turn at a particular locationare only experienced between 2 pm-4 pm on school days. As anotherexample, a sensor view may only be occluded during spring and summermonths, before foliage drops from a tree occluding the view. Suchtemporal patterns may be recognized using machine learning techniques.The timestamps 630 may also be used to update the information. Forexample, signals older than a predetermined threshold, such as severalmonths or days, may be deleted or may be attributed a lower weight whendetermining an associated level of difficulty. While the timestampinformation as shown in indicated by month:day:year::hour:minute:second,it should be understood that any other convention for indicating timemay be used.

The location information 640 may identify a location from which thesignal was transmitted. In the example shown, the location informationis indicated by a regional identifier, a street identifier, and asegment identifier, also other formats may be used. The regionalidentifier may include a value indicating a particular city, state,country, or other region. The street identifier may include a value orother indicia corresponding to a particular roadway which is dividedinto segments for purposes of analysis. Accordingly, the segmentidentifier indicates a particular segment of the street. It should beunderstood that the location format used in chart 600 is merely anexample, and that any of a variety of other formats may be used, such asgeographic coordinates (e.g., latitude, longitude, altitude) or otherdescriptors.

In some examples, the storage system 450 may also store otherinformation relating to difficulty of traversing a driving environment,such as a model based on the information regarding the difficultysignals. For example, the model may be a prediction model identifyingwhich road segments are likely to present difficulty, such as requiringtransition from autonomous mode to manual mode, at a given time.

The storage system 450 may also store information which can be providedto client computing devices for display to a user. For instance, thestorage system 450 may store predetermined distance information fordetermining an area at which a vehicle is likely to stop for a givenpickup or destination location. The storage system 450 may also storegraphics, icons, and other items which may be displayed to a user.

The vehicle 100 may use the information from the storage system 450 todetermine a route for the vehicle 100 to travel. For example, given apoint of origin and a destination, several different routes may beavailable to the vehicle 100. However, some routes may be more difficultto travel than others. The information regarding difficulty oftraversing the driving environments stored in the storage system 450 maybe used to determine which route will present the least amount ofdifficulty the given time. In some examples, such difficulty informationmay be considered in combination with other travel information, such astime, distance, tolls, etc. The time information may take into accounttraffic, speed limits, and other information affecting driving time.According to some examples, the difficulty information and other travelinformation may be weighted. For example, a preference for easier traveldespite longer travel time may be accommodated by adjusting weightsaccorded to difficulty information and time information, respectively. Apreference for a shortest distance despite potentially difficultconditions may be accommodated by adjusting the weights accorded todistance and difficulty, respectively.

Example Methods

In addition to the operations described above and illustrated in thefigures, various operations will now be described. It should beunderstood that the following operations do not have to be performed inthe precise order described below. Rather, various steps can be handledin a different order or simultaneously, and steps may also be added oromitted.

FIG. 7 presents a flow diagram 700 illustrating an example method ofselecting a route based on difficulty of driving environment. In block710, data is received, the data providing information regarding aplurality of road segments. The information may include signals from oneor more vehicles that have traversed the road segments, such as thesignals discussed above in connection with FIG. 6. The signals mayindicate a difficulty associated with traversing the road segment,wherein such difficulty is a result of the driving environment. Examplesof such difficulties include requiring manual takeover, requestingassistance from remote operators, unprotected maneuvers,limited/occluded/obstructed view, or the like. In some examples, thedifficulties may be signaled by user complaints, measured passengerdiscomfort, fault responses in which the vehicle can no longer operatenormally and needs to pull over and stop, etc. Passenger discomfort maybe measured by, for example, survey responses, changes in passengerheart rate, temperature, or other vitals, changes in the passenger'sseating position, adjustment of controls, etc.

In block 720, a difficulty value is determined for each of the pluralityof road segments based on the received information. For example, thedifficulty value may be a score or other value which may be used tocompare a level of difficulty associated with traversing a first roadsegment to a level of difficulty of traversing a second road segment.For example, each different type of difficulty indicated by the vehiclesignals may have a value. In some instances, the value for one type ofdifficulty may be higher than the value of another type. For example,the difficulty value associated with a request for assistance from aremote operator may be different than a difficulty value associated withan unprotected turn. In other examples, the difficulty values may bevaried based on other information, such that the values for two signalshaving a same type are different. These difficulty values may be used tocompute a difficulty value for the road segment. For example, thedifficulty values of the various signals may be factored into apredictive model, which predicts a difficulty that will be encounteredon a road segment if traveled. Such prediction may be time correlated,such that the predicted difficulty level may vary for a particular roadsegment depending on a time of day, week, season, or other temporalinformation. For a particular route segment, multiple difficulties maybe predicted to occur at a particular time of travel. Accordingly, thedifficulty value may be computed as a function of the multiple predicteddifficulties.

In block 730, a route cost for each of one or more possible routesbetween an origin and a destination is computed based on the difficultyvalues associated with the plurality of road segments. For example, theroute cost may be a total of the difficulty values for each road segmentalong the route. In some examples, the route cost may also factor inother travel variables, such as time, distance, traffic, etc.

In block 740, a route of travel for the autonomous vehicle is selectedbased on the computed route costs. Accordingly, a control system of theautonomous vehicle may cause the vehicle to travel the selected route.

FIG. 8 illustrates an example of multiple possible routes 810, 820, 830between an origin O and a destination D. The routes 810-830 are shownrelative to map information, such as the map information 200 of FIG. 2.

FIG. 9 illustrates the routes 810, 820, 830, wherein each route includesa plurality of road segments. For example, a first route 810 includessegments 1 a-1 h, second route 820 includes segments 2 a-2 f, and thirdroute 830 includes segments 3 a-3 g. The road segments may be definedfor each roadway indicated on a map, or only for each possible routebetween an origin and a destination. The road segments may divide theroadway or route into any number of discrete subportions for analysis.For example, additional smaller segments may be used for analysis withfiner granularity of potential difficulties, while fewer larger segmentsmay be used for faster analysis consuming less memory and computingresources. In some examples, the road segments may different in sizefrom one segment to the next. The road segments may be designated basedon any factors, such as absolute distance, distance between turns,estimated travel time, etc.

FIG. 10 illustrates the possible routes 810-830 showing relativedifficulty of each road segment. In this example, increased difficultyis represented by increased line thickness. Along route 810, segments 1a and 1 b have relatively low difficulty, while segment 1 c has a veryhigh associated level of difficulty. This may be attributed to thedifficulty of maneuvering the vehicle to make such a sharp left-handturn. Other factors may include visibility while making the turn andwhether the turn is unprotected. For example, if other vehiclestraveling along the roadway onto which the vehicle is turning do nothave to stop or yield while the vehicle is making the sharp left-handturn, the turn has increased difficulty. In the example shown, otherdifficult segments of route 810 include segment 1 f. This may beattributed to factors other than turning, such as high pedestriantraffic, a rough terrain, limited visibility, or the like.

Along route 820, segments 2 a and 2 b are also shown to have lowrelative difficulty. Because these segments corresponds to the sameportions of roadway as the segments 1 a and 1 b of route 810, it wouldbe expected that they have the same difficulty level. Segments 2 c-2 e,however, have a high level of associated difficulty, though still lowerthan that of segment 1 c. Such difficulty may be a result of having tomaneuver around vehicles attempting to make the turn of segment 1 c,waiting to make the unprotected left turn of segment 2 d, or any of aplurality of other possibilities. The difficulty of segment 2 e may be aresult of other driving conditions, such as an obstructed view (e.g.,from buildings, signs, foliage, parked cars, or any other objects),roadway construction, etc.

Along route 830, only segment 3 c is shown as having a level ofassociated difficulty. This difficulty level corresponds to that ofsegment 2 c, and is likely a result of the same driving conditions.While segment 3 d includes a left-hand turn, such turn may have a lowlevel of associated difficulty because it is a protected turn in thatvehicles traveling in opposing directions will stop pursuant to the stopsigns 251, 252.

FIG. 11 illustrates an example calculation of route cost for each ofpossible routes 810, 820, 830. For each route segment, a difficultyvalue attributed to that segment is listed. While in the example of FIG.10 the difficulty was indicated graphically, in FIG. 11 it isrepresented by a numerical value. It should be understood, however, thatany convention for indicating the difficulty level may be used.Consistent with FIG. 10, segment 1 c of the route 10 is indicated tohave a highest associated difficulty level. Other segments of relativelyincreased difficulty includes segment 1 f of route 810, segments 2 c-2 eof route 820, and segment 3 c of route 830. Because routes 820 do nothave a segment corresponding to h, those values are set to 0. The valuesfor each segment along the routes may be summed to derive a totaldifficulty value for the route. For example, route 810—has a totaldifficulty value of 20, route 820 has a total difficulty value of 18,and route 830 has a total difficulty value of 11. Accordingly, if solelyconsidering difficulty, route 830 may be the most desirable route with alowest route cost of 11. However, as mentioned above, in some examplesother factors may be considered in computing route cost, such asdistance, time, etc.

The foregoing examples describe an advantageous method of determining atravel route, because it expands traditional consideration of factorssuch as time and distance to further consider driving/riding experience.Passengers of autonomous vehicle may thus have an option of selecting asmoother or less risky drive, despite that it may take longer than otherroutes. This may be particularly preferable for passengers that want toread, work, or relax during the trip.

While a number of examples of different types of difficulty that may beencountered by a vehicle are described above, it should be understoodthat any number of further types of difficulty may also be factored intoa route cost computation. For example, on particular streets, garbagecollection may occur at predetermined times, such as between 7 am-8 amevery Tuesday and Friday. During garbage collection times, the slowmoving and frequently stopping garbage truck may present an obstacle fora vehicle traveling on that street. Moreover, presence of sanitationworkers collecting the garbage requires additional caution in operationof the vehicle traveling on that street. Accordingly, difficulty signalsfrom previous or historic drives along that street may indicate a numberof obstacles (e.g., garbage truck, garbage cans, sanitation workers,etc.) above a predetermined threshold. The difficulty signals may alsoindicate slower driving times, increased wait times, an increased numberof applications of a braking system, unexpected maneuvers (e.g.,swerving), etc. Any or all of these difficulty signals may be used incomputation of a cost for the particular street during the predeterminedgarbage collection times.

Just by way of further example, a given roadway may be near a school. Assuch, during school hours, and particularly during times of arrival atand dismissal from the school, an increased amount of pedestrians may bewalking on or crossing the given roadway. Accordingly, during thoseschool hours, an increased number of obstacles may be presented,requiring additional caution by vehicles traveling on the given roadway.The difficulty signals from previous vehicles having traveled the givenroad may indicate the presence of such additional obstacles during timescorresponding to school hours. Such additional obstacles may have insome cases resulted in disengagement or a request to a remote operatorfor assistance. These difficulty signals indicating the presence of suchadditional obstacles during the particular times on the given roadwaymay also be used to compute a cost for the traveling the given roadway.

Unless otherwise stated, the foregoing alternative examples are notmutually exclusive, but may be implemented in various combinations toachieve unique advantages. As these and other variations andcombinations of the features discussed above can be utilized withoutdeparting from the subject matter defined by the claims, the foregoingdescription of the embodiments should be taken by way of illustrationrather than by way of limitation of the subject matter defined by theclaims. In addition, the provision of the examples described herein, aswell as clauses phrased as “such as,” “including” and the like, shouldnot be interpreted as limiting the subject matter of the claims to thespecific examples; rather, the examples are intended to illustrate onlyone of many possible embodiments. Further, the same reference numbers indifferent drawings can identify the same or similar elements.

1. A method, comprising: receiving, by one or more computing devices,information from one or more vehicles configured to operate in anautonomous driving mode, the information identifying different types ofdifficulties encountered by the one or more vehicles while operatingalong one or more routes in the autonomous driving mode, wherein thedifferent types of difficulties include at least one of a sensordifficulty or a roadway condition impacting autonomous driving; storingthe information in memory along with metadata that identifies one ormore details regarding each one of the different types of difficulties;evaluating, by the one or more computing devices, the stored informationand the metadata in view of a plurality of route options; and selecting,by the one or more computing devices, a given one of the plurality ofroute options for use by a vehicle when operating in the autonomousdriving mode.
 2. The method of claim 1, wherein the information includesa vehicle signal identifier for each one of the different types ofdifficulties that identifies a specific signal sent by a respective oneof the one or more vehicles.
 3. The method of claim 2, wherein thevehicle signal identifier is either (i) included with the specificsignal sent by the respective vehicle, (ii) assigned by the one or morecomputing devices, or (iii) composed from a combination of informationfrom the respective vehicle and the one or more computing devices. 4.The method of claim 1, wherein the metadata identifies at least one of atime each one of the different types of difficulties was encountered, orlocation information identifying a location at which each one of thedifferent types of difficulties was encountered.
 5. The method of claim4, wherein the location information includes at least one of a regionalidentifier, a street identifier, or a road segment identifier.
 6. Themethod of claim 4, wherein the location information includesgeographical coordinates.
 7. The method of claim 1, wherein the metadatafurther includes a difficulty value for each one of the different typesof difficulties that indicates a level of difficulty imposed by adriving environment.
 8. The method of claim 7, further comprisingweighting the difficulty values based on temporal information associatedwith each one of the different types of difficulties.
 9. The method ofclaim 1, further comprising generating a prediction model identifyingwhich road segments are likely to present difficulty based on theinformation stored in the memory.
 10. The method of claim 1, wherein theinformation stored in the memory includes a signal type indicating atype of difficulty encountered by each of the one or more vehicles thattransmitted corresponding information.
 11. The method of claim 10,wherein the type of difficulty is one of an extended wait to make a turnor other maneuver, an unprotected turn, an occluded sensor view, a rapidbrake application, a limited sensor view, an unprotected lane crossing,or a driving difficulty resulting from a driving environment.
 12. Themethod of claim 10, wherein the type of difficulty is one of a usercomplaint, measured passenger discomfort, a fault response, a need for ahuman driver to take manual control, or a request for assistance from aremote operator.
 13. The method of claim 1, further comprisingtransmitting the selected route option to a particular vehicle foroperating in the autonomous driving mode.
 14. A method of operating avehicle in an autonomous driving mode, the method comprising:controlling, by one or more processors, driving operations of thevehicle in the autonomous driving mode; receiving, by the one or moreprocessors, sensor information from a perception system of the vehicleregarding objects or conditions in an external environment of thevehicle; generating, by the one or more processors, informationregarding a difficulty experienced while operating in the autonomousdriving mode, the information identifying at least one type ofdifficulty encountered by the vehicle while operating along one or moreroutes in the autonomous driving mode, wherein the at least one type ofdifficulty includes at least one of a sensor difficulty for a sensor ofthe perception system or a roadway condition impacting autonomousdriving; and transmitting the generated information to a remote system.15. The method of claim 14, wherein the generated information includes avehicle signal identifier for each one of the at least one type ofdifficulty.
 16. The method of claim 14, wherein the generatedinformation includes one or both of a time or a location at which eachof the at least one type of difficulty was encountered.
 17. The methodof claim 16, wherein the location includes at least one of a regionalidentifier, a street identifier, or a road segment identifier.
 18. Themethod of claim 14, wherein the at least one type of difficultyencountered by the vehicle is one of an extended wait to make a turn orother maneuver, an unprotected turn, an occluded sensor view, a rapidbrake application, a limited sensor view, an unprotected lane crossing,or a driving difficulty resulting from a driving environment.
 19. Themethod of claim 14, wherein the at least one type of difficultyencountered by the vehicle is one of a user complaint, measuredpassenger discomfort, a fault response, a need for a human driver totake manual control, or a request for assistance from a remote operator.20. The method of claim 14, further comprising: receiving, from theremote system, a routing signal based on difficulty information obtainedfrom one or more other vehicles; and controlling, by the one or moreprocessors, another driving operation of the vehicle in the autonomousdriving mode according to the received routing signal.